Implement custom protoc plugin to generate OTLP JSON class definitions by herin049 · Pull Request #4910 · open-telemetry/opentelemetry-python

herin049 · 2026-02-11T23:18:12Z

Description

This PR implements a custom protoc plugin to automatically generate OTLP JSON class definitions. Consensus was reached deciding that a custom protoc plugin would be the best route for adding support for OTLP JSON while minimizing the number of runtime dependencies (for additional context reference #4886). This PR is a refined version of the first part of a set of changes that are in #4902. Please view the draft PR for an overview of how these packages will be used.

Fixes #1003

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

tox run -e $(tox --listenvs | grep opentelemetry-protojson | tr '\n' ',')

Does This PR Require a Contrib Repo Change?

Yes. - Link to PR:
No.

Checklist:

Followed the style guidelines of this project
Changelogs have been updated
Unit tests have been added
Documentation has been updated

mmanciop · 2026-02-13T10:53:16Z

I am very much looking forward to this landing: protobuf is a pretty toxic dependency when automatically injecting Python SDKs with the OpenTelemetry Injector

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/analyzer.py

pmcollins · 2026-02-17T19:38:44Z

I think it would be valuable to add a test that validates the generated JSON by feeding it to google.protobuf.json_format.Parse. Maybe you could then compare dictionary representations of the original object with that of the resulting proto object for equality.

herin049 · 2026-02-17T23:58:47Z

I think it would be valuable to add a test that validates the generated JSON by feeding it to google.protobuf.json_format.Parse. Maybe you could then compare dictionary representations of the original object with that of the resulting proto object for equality.

Just to make sure I understand your suggestion properly, currently I have an integration test which runs the plugin to generate the ProtoJSON Python classes and tests the serialization/deserialization. Are you suggesting that we also generate the corresponding Protobuf definitions and then perform the JSON serialization using the ProtoJSON classes, load the JSON using the Protobuf definitions and then compare the attributes on the ProtoJSON and Protobuf classes?

herin049 · 2026-02-18T03:53:07Z

@pmcollins Since we mostly care about serialization for the SDK, I added tests to use google.protobuf.json_format.MessageToDict and verify that the contents are identical. Let me know what you think, and if there are another other test cases that might be useful to add.

pmcollins · 2026-02-18T15:08:17Z

Thanks @herin049. My suggestion is to simulate a Collector receiving JSON. In other words, end-to-end tests that create SDK objects typically sent to an exporter for emission (traces, metrics, and logs), serialize those objects into JSON using the functionality in this PR, send that JSON to our simulated Collector (google.protobuf.json_format.Parse), and make sure that the "Collector" produces data that matches what was fed into the other end of the pipe.

herin049 · 2026-02-18T15:33:27Z

@pmcollins I will be adding the opentelemetry-exporter-otlp-json-common and opentelemetry-exporter-otlp-json-http packages in a follow up PR to limit the size of this PR. The opentelemetry-exporter-otlp-json-common will be analogous to the existing opentelemetry-exporter-otlp-proto-common package and will include the logic to convert SDK objects to the ProtoJSON objects in the generated opentelemetry-proto-json package added in this PR. Do we want to add the roundtrip tests you are describing into the opentelemetry-exporter-otlp-proto-common package, and defer this to the next PR? I am open to including them in this PR as well, just not sure where exactly we would want these tests to live. Feel free to look at the draft PR #4902 for additional context.

pmcollins · 2026-02-19T15:17:47Z

Makes sense @herin049 -- can be in a subsequent PR.

herin049 · 2026-02-27T15:52:36Z

@aabmass I have rewritten the commit history to separate out the addition of the opentelemetry-codegen-json package an the auto generated opentelemetry-proto-json package with a final commit to integrate these packages into the rest of the repo. Hopefully this makes reviewing a bit easier, let me know if you would prefer that I break down these changes further.

aabmass

It's a bit large LGTM. I will look over the generated code too

codegen/opentelemetry-codegen-json/README.rst

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/runtime/json_codec.py

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/plugin.py

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/types.py

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/writer.py

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/generator.py

codegen/opentelemetry-codegen-json/tests/conftest.py

codegen/opentelemetry-codegen-json/tests/test_serdes.py

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/runtime/__init__.py

aabmass · 2026-03-05T18:50:18Z

opentelemetry-proto-json/src/opentelemetry/proto_json/collector/logs/v1/logs_service.py

+    def to_json(self) -> builtins.str:
+        """
+        Serialize this message to a JSON string.
+
+        Returns:
+            JSON string
+        """
+        return json.dumps(self.to_dict())


Would it be reasonable to put this in a base class to try to reduce the code size a bit? Maybe there are downsides, lmk

I've opted to add a @json_serde class decorator

opentelemetry-proto-json/src/opentelemetry/proto_json/collector/logs/v1/logs_service.py

aabmass · 2026-03-05T18:59:02Z

pyproject.toml

 typeCheckingMode = "standard"
 pythonVersion = "3.9"

 include = [


Can we typecheck the plugin code here, and ideally even the generated code as a sanity check?

Was it not possible to add opentelemetry-proto-json too?

MikeGoldsmith

Looks good to me outside of what @aabmass raised. Found one minor typo 👍🏻

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/types.py

…lity

aabmass · 2026-03-09T19:38:54Z

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/runtime/json_codec.py

+T = typing.TypeVar("T")
+
+
+def json_serde(cls: type[T]) -> type[T]:


Do static analysis tools and pyright know about the to_json() on the resulting class?

Yeah, I don't think it is possible to properly type this with current versions of Python. I'm just going to switch to using inheritance

aabmass · 2026-03-09T19:39:53Z

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/runtime/json_codec.py

+        Deserialize from a JSON string or bytes.
+        """
+        # pylint: disable-next=no-member
+        return cls_inner.from_dict(json.loads(data))  # type: ignore


You could probably fix this by using a Protocol.

aabmass · 2026-03-09T19:54:20Z

pyproject.toml

 typeCheckingMode = "standard"
 pythonVersion = "3.9"

 include = [


Was it not possible to add opentelemetry-proto-json too?

…on methods

MikeGoldsmith

Looks good to me. Needs the public symbols reviewing but otherwise 🚀

lzchen · 2026-03-11T16:21:22Z

opentelemetry-proto-json/README.rst

+``../scripts/proto_codegen_json.sh``. Then run the script and commit the changes
+as well as any fixes needed in the OTLP exporter.
+
+.. _opentelemetry-codegen-json: https://github.com/open-telemetry/codegen/opentelemetry-codegen-json


Nit: Shouldn't this link to this repository?

herin049 requested a review from a team as a code owner February 11, 2026 23:18

herin049 force-pushed the feat/protoc-otlp-json-plugin branch from f66380d to a9d10c7 Compare February 12, 2026 04:09

xrmx added this to Python PR digest Feb 12, 2026

xrmx moved this to Ready for review in Python PR digest Feb 12, 2026

aabmass reviewed Feb 17, 2026

View reviewed changes

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/analyzer.py Outdated Show resolved Hide resolved

herin049 force-pushed the feat/protoc-otlp-json-plugin branch from 4d4119c to 4bce86a Compare February 18, 2026 03:49

herin049 requested a review from aabmass February 24, 2026 17:29

herin049 force-pushed the feat/protoc-otlp-json-plugin branch from 4bce86a to 05b459f Compare February 27, 2026 15:49

aabmass approved these changes Mar 5, 2026

View reviewed changes

aabmass reviewed Mar 5, 2026

View reviewed changes

MikeGoldsmith reviewed Mar 6, 2026

View reviewed changes

codegen/opentelemetry-codegen-json/src/opentelemetry/codegen/json/types.py Outdated Show resolved Hide resolved

herin049 added 7 commits March 7, 2026 23:38

add opentelemetry-codegen-json package

72116ed

add opentelemetry-proto-json package

b8ec247

update project files to incorporate new packages

4956bbb

add 'Private :: Do Not Upload' to codegen pyproject.toml classifiers

dbbc98d

add @json_serde decorator and update runtime utility file name

9a7307e

update docstring

fd49bc2

add typechecking and linting for json codegen package

3582fb2

herin049 force-pushed the feat/protoc-otlp-json-plugin branch from 5052584 to 3582fb2 Compare March 8, 2026 04:41

herin049 added 3 commits March 7, 2026 23:52

update to use 'path.as_posix()' for Windows compatability

f22dd5f

add 'importlib.invalidate_caches()' to generate code fixture

5dd7918

add USERNAME=runner env var to github workflows for Windows compatabi…

8da2e56

…lity

herin049 added 2 commits March 8, 2026 00:11

update workflows

d4ffe02

update tox.ini

2bf60bc

herin049 requested a review from aabmass March 8, 2026 21:04

aabmass approved these changes Mar 9, 2026

View reviewed changes

update to use inheritence for common JSON serialization/deserializati…

c9cae12

…on methods

herin049 force-pushed the feat/protoc-otlp-json-plugin branch from ab4bed2 to c9cae12 Compare March 10, 2026 19:43

MikeGoldsmith approved these changes Mar 11, 2026

View reviewed changes

lzchen reviewed Mar 11, 2026

View reviewed changes

lzchen approved these changes Mar 11, 2026

View reviewed changes

		T = typing.TypeVar("T")


		def json_serde(cls: type[T]) -> type[T]:

Conversation

herin049 commented Feb 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

How Has This Been Tested?

Does This PR Require a Contrib Repo Change?

Checklist:

Uh oh!

mmanciop commented Feb 13, 2026

Uh oh!

Uh oh!

pmcollins commented Feb 17, 2026

Uh oh!

herin049 commented Feb 17, 2026

Uh oh!

herin049 commented Feb 18, 2026

Uh oh!

pmcollins commented Feb 18, 2026

Uh oh!

herin049 commented Feb 18, 2026

Uh oh!

pmcollins commented Feb 19, 2026

Uh oh!

herin049 commented Feb 27, 2026

Uh oh!

aabmass left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aabmass Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MikeGoldsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MikeGoldsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

herin049 commented Feb 11, 2026 •

edited

Loading

aabmass Mar 5, 2026 •

edited

Loading